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(54) Method for editing image information with aid of computer and editing system 



(57) There is provided an editing method and an 
editing system for conducting editing processing on a 
video information material containing sound information 
and moving picture information with the aid of a compu- 
ter. Information of the video information material is 
stored in a storage device (506). The video information 
is read out from the storage device (506). The read out 
video information (102, 103. 104, 121, 122. 126, 607, 
934) is displayed on the screen of a display device. An 
editing position of the video information is displayed on 
the video information of the screen according to a com- 
mand from a user. Upon a command from the user, the 
editing position (106. 107. 129. 130. 131. 132. 905) is 
altered. Editing processing (110, 610, 611, 1305) spec- 
ified by the user is conducted on video information 
located in the displayed editing position. And edited 
video information (104, 106, 934) is displayed on the 
screen. 
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Description 

BACKGROUND OF THE INVENTION 

The present invention relates to editing processing 
of moving images such as video pictures and sounds, 
and in particular to an image editing method and an 
image editing apparatus suitable for creation of televi- 
sion broadcast programs and video programs, and to a 
medium having a program recorded thereon to make a 
computer execute image editing processing operation. 
In recent years, methods for creating tele 

vision broadcast programs and video programs, 
enhanced by using computers have been rapidly 
advanced. Instead of video editing techniques of using 
a video tape recorder and repeating the fast feed and 
rewinding operation, editing is conducted by using 
images stored in a hard disk which is inexpensive and 
easy to handle and preserve. Moving image editing 
apparatuses using such techniques of so-called direct 
scheme have recently attracted attention. 

In JP-A-4-207877, there is disclosed a moving 
image managing apparatus for displaying moving 
images on a screen in a hierarchical structure of a tree 
form including cuts and scenes and conducting editing 
by taking a scene or a cut as the unit 

Herein, the term "moving image" means image 
information which contains pictures of a plurality of 
frame units or sound information or both of them 
arranged respectively independently in time series and 
which can be displayed as moving images. 

As for the image editing work, work to be decided 
by using the sensitivity of an editor is main work in any 
case. As for the image editing method, an apparatus of 
a scheme of conducting image editing by using the dis- 
play for manipulating video pictures on the image edit- 
ing screen, i.e., so-called icon display (hereafter 
referred to as icon) in association with image informa- 
tion is used. 

In the conventional technique of such a image edit- 
ing apparatus, effect editing is conducted between a 
desired cut (a series of a plurality of frame unit pictures) 
included in time serial video picture materials (moving 
pictures) or a scene formed by a plurality of cuts and 
another desired cut or another scene. FIGS. 2 and 3 
show examples of screen display obtained at this time. 
That is. in the effect editing, a picture is switched over to 
another one while the contents of the picture is 
changed. 

First of all, FIG. 2 shows an example of display of a 
screen for the computer aided editing work. FIG. 3 is a 
different example of display of a screen for the computer 
aided editing work. Both of these editing works use rep- 
resentative software for picture editing which is com- 
mercially available. 

The general configuration of the editing apparatus 
used at this time is a system based upon a computer. To 
be more concrete, it is a typical and general-purpose 



configuration having a central processing unit, a mem- 
ory and various devices operated by an operator. 

FIG. 4 shows an example of an image editing sys- 
tem. The image editing system has a computer 401. To 
5 the computer 401 , a monitor 407 including a display 
screen having a size of, for example. 19 inches is con- 
nected. 

A video reproduction unit (VTR) 402 can reproduce 
any video information (image information) of the NTSC 
io system, PAL system, or any other format system. The 
video information reproduced by this video reproduction 
unit 402 is sent to a video capture/ frame storage unit 
403. and converted into digital information and stored 
therein. 

15 The video information thus stored in the video cap- 
ture/ frame storage unit 403 is then supplied to a disk 
accelerator 405 via an expander/compressor 404. 

This expander/compressor 404 functions to com- 
press and expand video information. An expand - 

20 ing/compressing processor using a JPEG (Joint 
Photographic coding Experts Group) chip having the 
highest rate at the present time is often used. 

The disk accelerator 405 functions to transmit the 
video information sent from the video reproduction unit 

25 402 to the video capture/ frame storage unit 403 and 
stored therein to at least one disk storage unit 406, or 
functions to receive information therefrom. 

By using control signals 408. 409 and 41 0. the com- 
puter 401 controls components of the image editing 

30 apparatus, i.e., the video capture/ frame storage unit 
403, the expander/compressor 404, and the disk accel- 
erator 405. 

For the purpose of the editing work and other works 
to be then executed, the video information stored in the 

35 disk storage unit 406 can be read out from the the disk 
storage unit 406, and read into the video capture/frame 
storage unit 403 via the disk accelerator 405 and the 
expander/compressor 404 by using the control signals 
408. 409 and 41 0 supplied from the computer 401 . 

40 In accessing desired video information, the appara- 
tus shown in FIG. 4 has a feature in that it can "immedi- 
ately" access (i.e., randomly access) the video 
information unlike the conventional sequential file appa- 
ratus needing the rewinding operation of the video tape 

45 in principle. As described above, therefore, the tech- 
nique of the direct scheme is used. 

The aforementioned FIGS. 2 and 3 show first and 
second examples of a screen for the editing work dis- 
played on the monitor 407, respectively. In FIGS. 2 and 

so 3, vertical lines denoted by numerals 210 and 313 are 
figures for indicating the reference position at the time of 
editing work, and they are called current position bars. 

Numerals 302 and 303 on the screen of FIG. 3 
denote time lines for relatively indicating the position 

55 and the range of a cut or a scene to be edited on the 
basis of the position of the above described current 
position bar. For each picture information (image infor- 
mation) to which the cut or scene to be edited belongs, 
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a track for time line display is assigned and a corre- 
sponding track name is provided. 

On the screen of FIG. 2, numeral 203 denotes a 
time tine for relatively indicating the position and the 
range of a cut or a scene after the editing. On the screen 5 
of FIG. 3, numeral 304 denotes a time line indicating the 
range of synthesis when pictures of the two time lines 
302 and 303 are combined and edited. 

Each of numerals 201 (FIG. 2) and 301 (FIG. 3) 
denotes a time line window. On the window, the above io 
described time line is displayed. The time lines 203, 302 
and 303 can be moved in a scroll manner by screen 
manipulation. 

Numeral 205 denotes a program window. In order 
to confirm a scene of the image (picture) of the time line is 
203 displayed in the time line window 201. its image is 
displayed in the program window 205 according to 
image information (video information) read out from the 
disk storage unit 406. A group of buttons for directing 
mark-in and mark-out positions which are reproduction 20 
start and reproduction stop positions of the image dis- 
playing with a mouse (not illustrated) which is a direc- 
tion device connected to the computer 401 of FIG. 4 are 
also displayed. 

Numerals 206 and 306 denote windows called 25 
working bin window or project window, for displaying 
video information (image information) stored in the disk 
storage unit 406. mainly edited image information. 

For conducting effect editing, a window 305 for dis- 
playing a list of various video effect patterns and an 30 
effect setting window 312 for setting the kind of the 
effect or the effect quantity, or a video effect panel win- 
dow 207 having the functions of these two windows 
together are provided. 

The effect editing is editing for conducting special 35 
effect processing such as a wipe effect or a dissolve 
effect on an image. In the process of transition from a 
picture A to a picture B, the wipe effect is image 
processing (picture processing) of eliminating the pic- 
ture A from the screen in a certain changing pattern as 40 
if it wipes off the picture A and instead causing a differ- 
ent picture B to appear in an area from which the picture 
A has been eliminated. In the process of transition from 
a picture A to a picture B. the dissolve effect is picture 
processing of causing a picture B to gradually appear 45 
while causing the luminance of the picture A to gradu- 
ally fall and disappear. In the effect editing, other special 
effect processings other than the wipe effect and the 
dissolve effect may be also included. 

Operation required for the effect editing will now be so 
described in more detail by referring to FIGS. 2 and 3. 

( Case of FIG. 2 > 

For each scene or cut displayed on the working bin 55 
window 206. two out of different video materials are first 
selected by manipulation of the mouse and arranged on 
the time line 203 in the order of coupling. 



Subsequently, the bar 210 is moved to the position 
of the coupling of the two video materials by the mouse. 
As a result, the effect editing is started in the position of 
the moved bar 210. 

In other words, editing work desired by the user is 
conducted by manipulating a menu bar 208 for selecting 
an effect kind, a window 209 in which an effect time can 
be set by means of a numerical input from a keyboard 
(not illustrated) connected to the computer 401 in the 
same way as the mouse, and sliders allowing the user 
to set other values, such as the width of the boundary of 
the vertical wipe in the scene transition from the left to 
the right, or the gradation quantity. 

At this time, a plurality of still pictures 202 after edit- 
ing are arranged and displayed on the time line window 
201 at predetermined frame intervals. Thereby, it is pos- 
sible to know the state of the effect of the edited scene 
or cut. As for the number of frame in the predetermined 
frame intervals, variable setting is possible. This setting 
is conducted by selecting a predetermined menu from a 
menu bar included in the program window 205. 

<Case of FIG. 3) 

From video materials differing in scene or cut dis- 
played in the project window 306. two video materials 
are selected and disposed on the time lines 302 and 
303 as role A and role B, respectively. 

In the case where two scenes or cuts are subject to 
synthetic editing, the terms "roll A" and "roll B" are used 
to respectively represent the two scenes or cuts while 
making a distinction between them. The synthetic edit- 
ing in this case is also referred to as "AB role editing." 

In order to set the effect time, the time line 303 is 
then moved by the mouse so as to attain a predeter- 
mined overlap width with respect to the time line 302 in 
the direction of the axis (time axis) of abscissas. 

If any one pattern is selected by the user out of var- 
ious effect patterns displayed in the video effect pattern 
display window 305, then the effect setting window 312 
is displayed, and an effect pattern symbol is displayed in 
the range of the above described overlap on the time 
line 304. 

On the effect setting window 312, the user inputs or 
vary the effect setting quantity, and conducts the 
desired editing work 

At this time, still pictures before editing are dis- 
played respectively on the time lines 302 and 303 in the 
time line window 301 at predetermined frame intervals. 
This setting of the number of frame intervals is con- 
ducted in a slider 315. 

In the editing system described by referring to 
FIGS. 2. 3 and 4, it cannot be said that sufficient consid- 
eration is not given to the display required for the image 
editing, and in particular to its kind selection. The image 
editing work cannot be simplified. The editing system 
has problems in making effective image editing possi- 
ble. 
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In other words, first in the example of FIG. 2, the 
editing work is conducted by the numerical input from 
the keyboard while the operator is confirming the con- 
tents of the edited effects by means of the still pictures 
202 displayed at the predetermined frame intervals. At 
this time, however, simultaneous display of pictures cor- 
responding to the roles A and B which is effective in 
facilitating the AB role editing is not conducted. There- 
fore, the operator cannot determine the start point and 
end point of a scene transition at the time of scene syn- 
thesis while confirming the images obtained before edit- 
ing. 

Subsequently, in the example of FIG. 3, video mate- 
rials of different scenes displayed in the project window 
306 are respectively selected and moved onto the time 
lines 302 and 303. The editing work is thus conducted 
while displaying the pictures of the roles A and B. There- 
fore, the problems offered in the case of FIG. 2 are not 
posed. Unlike FIG. 2, however, there is no display corre- 
sponding to the still picture display (still pictures) 202 
after the effect editing. Therefore, the user cannot con- 
firm the scene transition state obtained when the effect 
editing has been conducted. 

In the above described two examples, therefore, the 
start points and end points of the scene transitions of 
the role A and role B to be edited are confirmed by the 
picture display or by observing the picture of the role AB 
resulting from the synthetic editing. Thus there remain 
problems that they cannot be confirmed quickly by the 
picture display of the time line window alone and dis- 
plays must be switched over. 

In general, moving image are accompanied by 
sound information. In this case, editing on sounds also 
becomes necessary. Editing modes of sounds at this 
time are roughly classified into the following two kinds. 

A first mode is the so-called overlap editing. When 
splicing a certain scene of a video picture B to a certain 
scene of a video picture A, sound A is gradually attenu- 
ated in the splicing place and sound B is gradually aug- 
mented. These states are overlapped. 

This overlap editing is used for the purpose of pre- 
venting the unnaturalness caused by a sudden change 
of sound at the splicing point and making a smooth tran- 
sition between scenes possible. 

A second mode is the so-called mixing editing. 
Apart from sounds accompanying the video images, 
new sounds are added to the video images in the mixing 
editing. 

This mixing editing is used when BGM (background 
music) or the like is desired to be accompanied with 
video pictures. 

According to the method which has become the 
mainstream in conventional broadcast program produc- 
tion sites when conducting such editing, two VTRs are 
prepared and reproduction outputs of them are inputted 
to a mixer. While listening to the reproduced sound, the 
editor manipulates the mixer. While the editor is deter- 
mining the degree of attenuation and the degree of aug- 



mentation of each sound during the reproduction, the 
editing is conducted. 

The above described conventional techniques have 
problems that sufficient consideration is not given to 

5 facilitating the editing work and the editing work requires 
high skill and a great deal of labor and time. 

In the editing according to the conventional tech- 
nique as described above, two VTRs are prepared and 
reproduced outputs of them are inputted to the mixer. 

10 The editor must manipulate the mixer and determine the 
degree of attenuation and the degree of augmentation 
during the reproduction while listening to the sounds. 
Therefore, high skill is required for the work. In addition, 
in the case where modification or alteration of editing is 

75 to be conducted, it is necessary to rewind a VTR once 
and try again from the beginning. As a result, a great 
deal of labor and time are required. 

In the so-called nonlinear editing heretofore used, 
the editing mode using the time line scheme forms the 

20 mainstream. 

According to this scheme, editing is advanced by 
inserting materials of the video picture and sound with 
mouse manipulation in a window of strip-like display 
with the time axis direction replaced by the horizontal 

25 direction of the display screen. 

In this time line scheme as well, however, both the 
video picture and sound are displayed by only strips 
having lengths according to the time lengths, respec- 
tively. This results in a problem that conducting the work 

30 while confirming the video pictures is difficult when 
determining the edit point in the overlap editing and 
determining the mixing start point in the mixing editing. 

Examples of the computer aided video and audio- 
editing systems are disclosed in the user's guide manu- 

35 als of Adobe Premiere version 4.0 (Adobe Systems 
Incorporated. 1994) and MEDIA SUITE PRO Version- 
1 .0 (Avid Technology, Inc., 1994). 

SUMMARY OF THE INVENTION 

40 

The present invention provides an editing method 
and an editing system capable of conducting the effect 
editing of images that like pictures or the sound or the 
both of them simply and rapidly by manipulation on a 

45 computer graphical user interface screen, and provides 
a recording medium storing a program for executing the 
editing processing. 

The present invention provides an editing method 
and an editing system for conducting editing processing 

so on an image information material containing sound 
information and moving picture information with the aid 
of a computer. 

In the editing method and the editing system 
according to the present invention, information of the 

55 image information material is stored in a storage device. 
The image information is read out from the storage 
device. The read out image information is displayed on 
the screen of a display device. An editing position of the 
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image information is displayed on the display of the 
image information of the screen according to a com- 
mand from a user. Upon a command from the user, the 
editing position is altered. Editing processing specified 
by the user is conducted on image information located 5 
in the displayed editing position. And edited image infor- 
mation is displayed on the screen. 

Therefore, the user can conduct editing processing 
while confirming both the editing material and the edit- 
ing result simultaneously on the screen. 10 

According to an embodiment of the present inven- 
tion, editing of the moving picture information and/or the 
sound information can be conducted on the screen of 
the display device. According to another embodiment of 
the present invention, the display of relating sound infor- is 
mation is altered so as to be linked with the editing of 
the moving picture information. According to a different 
embodiment, reduced (contracted) moving picture infor- 
mation are created by reducing pixels of the moving pic- 
ture information, and the contracted moving pictures 20 
thus created are stored in the storage device. From the 
contracted moving picture information read out from the 
storage device, contracted moving pictures of a film 
image are created. The moving pictures of the film 
image thus created are displayed on the screen as the 25 
editing materia!. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a diagram showing an example of an effect 30 
editing screen display according to an embodiment 
of the present invention; 

FIG. 2 is a diagram showing an example of a screen 
display of effect editing; 

FIG. 3 is a diagram showing another example of a 35 

screen display of effect editing; 

FIG. 4 is a block configuration diagram showing an 

example of a moving image editing apparatus 

according to a conventional technique; 

FIG. 5 is a block configuration diagram of an 40 

embodiment of a moving image editing apparatus 

according to the present invention; 

FIG. 6 is a diagram showing an example of a tree 

structure image on a display screen according to an 

embodiment of the present invention; 45 

FIG. 7 is a data configuration diagram of a picture 

information file in an embodiment of the present 

invention; 

FIG. 8 is a data configuration diagram of a hierar- 
chical structure management information file in an so 
embodiment of the present invention; 
FIG. 9 is a fbw chart of a program for controlling 
effect editing in an embodiment of the present 
invention; 

FIG. 10 is a diagram showing an example of a hier- ss 
archical structure image of picture information 
using a combination of representative pictures in an 
embodiment of the present invention; 



FIG. 11 is a diagram showing another example of a 
hierarchical structure image of picture information 
using a combination of representative pictures in an 
embodiment of the present invention; 
FIG. 12 is a diagram showing a display state before 
transition to an effect editing screen display in an 
embodiment of the present invention; 
FIG. 1 3 is a block diagram showing an example of a 
moving image editing apparatus used in an embod- 
iment of the present invention; 
FIG. 14 is a schematic diagram of a display screen 
before a display for editing in an embodiment of the 
present invention; 

FIG. 15 is a diagram showing operation conducted 
by a moving image editing apparatus used in an 
embodiment of the present invention; 
FIG. 16 is a schematic diagram showing an exam- 
ple of an editing screen display according to an 
embodiment of the present invention; 
FIGS. 17A and 17B are examples of an editing 
screen display according to an embodiment of the 
present invention; 

FIG. 18 is a diagram for explaining overlap sound 
editing in an embodiment of the present invention; 
FIG. 19 is a schematic diagram of a display screen 
after overlap sound editing in an embodiment of the 
present invention; 

FIG. 20 is a schematic diagram of a reproduction 
display screen for confirming the editing state in an 
embodiment of the present invention; 
FIG. 21 is a schematic diagram showing another 
example of an editing screen display according to 
an embodiment of the present invention; 
FIG. 22 is a diagram for explaining overlap sound 
editing in the case where a special effect such as 
wipe or dissolve has been applied to video pictures, 
in an embodiment of the present invention; 
FIG. 23 is a diagram for explaining overlap sound 
editing in the case where a special effect such as 
wipe or dissolve has been applied to video pictures, 
in an embodiment of the present invention; 
FIG. 24 is a schematic diagram showing still 
another example of an editing screen display 
according to an embodiment of the present inven- 
tion; 

FIG. 25 is a diagram for explaining the operation of 
sound mixing editing in an embodiment of the 
present invention; 

FIG. 26 is a schematic diagram of a display screen 
after sound mixing editing in an embodiment of the 
present invention; 

FIG. 27 is a flow chart showing the procedure of the 
overlap editing shown in FIGS. 17A and 17B; 
FIG. 28 is a flow chart showing the procedure of 
wipe editing shown in FIG. 21 ; and 
FIG. 29 is a flow chart showing the procedure of 
mixing editing shown in FIG. 24. 
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DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

Hereafter, an image editing apparatus according to 
the present invention will be described in more detail by 5 
referring to illustrated embodiments. 

FIG. 5 shows an example of a hardware configura- 
tion in an embodiment of the present invention. The 
hardware configuration includes a CPU 501 for process- 
ing various controls, a monitor 508 for displaying vari- 10 
ous kinds of information, such as image information of a 
scene and a cut used for image editing and a hierarchi- 
cal structure (tree structure) representing the editing 
state, a memory 502 for storing various control pro- 
grams of the CPU 501, a storage device, such as a is 
magnetic storage device, 506 for storing image informa- 
tion, sound information, and editing information relating 
to the information, a frame buffer memory 507 for stor- 
ing images to be displayed on the monitor 508, a scene 
change point detector 503 for detecting a scene change 20 
point between cuts, a mouse (or keyboard) 509 serving 
as an input device, a compressor/expander portion 51 1 
for compressing picture information supplied from a 
video tape recorder (hereafter referred to as VTR) 505 
to generate compressed picture data and expanding 25 
compressed picture data to reproduce picture informa- 
tion equivalent to the picture information supplied from 
the VTR 505. 

From the image information stored in the magnetic 
storage device 506, contracted (reduced) pictures for 30 
high speed display obtained from the original image 
material by reducing the information content to, for 
example, information reduced by 1/64 are generated 
and stored in the magnetic storage device 506 under 
the control of the CPU 501 . 35 

The CPU 501 , the memory 502, the scene change 
point detector 503, a video interface 504, the magnetic 
storage device 506, and the frame buffer memory 507 
are connected to a bus 510. As a result, respective 
components are controlled by access from the CPU 40 
501. 

In the magnetic storage device 506, the following 
various kinds of information are stored: 

(a) frame numbers associated with frames in a 45 
series of pictures, or frames at change points 
detected by the cut change point detector 503; 

(b) name of file in which contracted picture data for 
high speed display have been filed; 

(c) name of file in which compressed picture data so 
have been filed; 

(d) contracted picture data for high speed display; 

(e) compressed picture data; 

(f) hierarchical number representing a hierarchical 
step number for a plurality of scenes or cuts; ss 

(g) cut number serially provided in the order of cut 
in a plurality of video image materials; 

(h) sequence number representing the order, in a 



scene, of each of a plurality of cuts forming the 
scene; 

(i) identification information representing whether 
each scene or cut should be displayed in a hierar- 
chical structure when the hierarchical structure is 
displayed on the monitor 508; 

As the magnetic storage device 506, a magnetic 
disk, an optical disk, or a different recording medium 
such as a magnetic-optical disk storage device may be 
used, or a remote file made accessible via a communi- 
cation network (LAN or a network using an RS-232C 
interface) may be used. 

Operation of an embodiment of the present inven- 
tion thus configured will now be described. 

First of all, in response to a command from the 
mouse (or keyboard) 509, a video signal which is a 
video image information source is outputted from video 
tape mounted on the VTR 505. Via the video interface 
504 and the bus 510, image information according to 
the video signal is registered in the magnetic storage 
device 506. 

According to the video signal outputted from the 
VTR 505 at this time, image information of each frame 
unit is supplied to the scene change point detector 503. 
In the scene change point detector 503. the image infor- 
mation is encoded and scene change points in the 
encoded image information are detected. Information 
concerning the scene change points is also registered 
in the magnetic storage device 506. 

As a result, image information including a frame 
number corresponding to a frame of a scene change 
point of a detected scene change, a contracted picture 
for high speed display and a file name in which the pic- 
ture has been filed, and a compressed picture and a file 
name in which the picture has been filed is stored in the 
magnetic storage device 506. 

In other words, the video signal outputted from the 
VTR 505 is subjected to data compression in the com- 
pressor/expander 511, and the compressed picture 
data are also registered in the magnetic storage device 
506. 

Ihe contracted pictures for high speed display are 
provided so as to be read out at high speed from the 
magnetic storage device 506 when contents of the 
whole of moving images formed by a plurality of frames 
or a scene are to be confirmed or the result of editing is 
to be confirmed. When the video signal is outputted 
from the VTR 505, the contracted pictures for high 
speed display are also generated on the basis of the 
video signal under the control of the CPU 501 and reg- 
istered in the magnetic storage device 506. 

The "contracted pictures for high speed display" are 
pictures contracted in picture size and reduced in 
number of pixels as compared with the original pictures 
in order to display a large number of cut pictures repre- 
sentative of each scene on the screen of the monitor 
508 immediately in response to editing manipulation. In 
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the editing system of the present invention, both a mate- 
rial picture to be edited and a picture completed in edit- 
ing are displayed on a single screen substantially in an 
instant in response to a user's command. For that pur- 
pose, the information volume (such as the number of 
pixels or the number of scanning lines) of the original 
picture information is thinned out by a predetermined 
amount. Thus the information volume is reduced so as 
to allow high speed readout from the magnetic storage 
device 506. In addition, the size of the cut picture is con- 
tracted so that the display screen of the monitor may 
accommodate a required number of cut pictures. In this 
way, contracted pictures for high speed display are pre- 
pared beforehand. The contracted pictures for high 
speed display are produced by the CPU 501 by con- 
ducting processing for reducing the information volume 
of the original picture information of various video sys- 
tems such as the NTSC system or the PAL system. 

When it is desired to confirm the contents of the 
above described scene, the pictures for high speed dis- 
play thus contracted are used to display the desired 
scene on a high-speed display window 602 shown in 
FIG. 6 in response to a command using the mouse (or 
keyboard) 509. 

The window display of FIG. 6 is displayed on the 
screen of the monitor 508 before the effect editing 
according to the embodiment of the present invention is 
started. 

For taking out a predetermined frame picture from 
picture data subjected to data compression, it is neces- 
sary to conduct expansion processing as is generally 
known. If the compressor/expander is not present as 
hardware, therefore, it is necessary for the CPU 501 
and the compressor/expander 51 1 to conduct picture 
processing and effect the compression by using the 
algorithm of the JPEG system, for example. In the case 
of data compression/expansion processing using soft- 
ware, a long processing time is needed. 

In this embodiment, contracted picture data for high 
speed display are recorded in the magnetic storage 
device 506 beforehand. Even if the compres- 
sor/expander for conducting the compression/expan- 
sion processing at high speed is not present as 
hardware, therefore, the contracted picture data for high 
speed display can be read out without requiring the 
processing time for expansion. In other words, expan- 
sion processing is unnecessary in the case where the 
contracted picture for high speed display are repro- 
duced. 

If the operator commands readout of the picture 
information for high speed display registered in the 
magnetic storage device 506 by using the mouse (or 
keyboard) 509, the CPU 501 reads out only first frames 
of respective cuts representing scene change points of 
pictures. On a display screen 600 of the monitor 508, a 
window 603 for taking a look at cuts as shown in FIG. 6 
is thus displayed. 

With respect to this display, the operator informs the 



CPU whether the scene change point detected by the 
scene change point detector 503 is proper for the con- 
tents of the image and of scene partitions by using the 
mouse (keyboard) 509, 

5 The CPU 501 provides each of cuts judged to be 

proper as a scene change point of images stored in the 
magnetic storage device 506 with a cut number. 

If image editing or the like is commanded by the 
operator via the mouse (or keyboard) 509, tree structure 

to management information as shown in FIGS. 7 and 8 is 
read out from the magnetic storage device 506 or the 
memory 502. 

FIG. 10 represents frame pictures (frame images) 
forming moving pictures recorded in the magnetic stor- 

15 age device 506 by using a hierarchical structure. In the 
system of the present embodiment, this hierarchical 
structure is displayed on the monitor screen. The term 
"cut" is defined as a set of frame pictures (frame 
images) shot by a TV camera from the start of a shoot- 

20 ing operation until the end thereof. A "scene" is formed 
by a set of a plurality of cuts obtained by shooting a cer- 
tain fixed shoot subject. Furthermore, a plurality of 
scenes can be collected by a specific shoot theme. ~ 
Video sources can be arranged as a hierarchical struc- 

25 ture including a plurality of cuts, a scene having a col- : 
lection of a certain number of cuts, and a shoot theme 
or a title having a collection of a certain number of : 
scenes. Each of cuts and scenes is provided with a spe- 
cific number. 

30 FIG. 10 shows an example of a treelike hierarchical 
structure of recorded video sources. It is now assumed 
that the shoot theme is, for example, "tropical island.*' In 
the recorded video sources, a cut 1 obtained by shoot- . 
ing fish in the sea around an island, a cut 2 obtained by 
35 shooting a coastal landscape, a cut 3 obtained by 
shooting birds inhabiting a forest, and a cut 4 obtained 
by shooting a flower are included. The cut 1 is formed by 
1st through 90th frames, and the cut 2 is formed by 91 st 
through 150th frames. Video pictures of the cut 1 and 
40 the cut 2 are collected as a scene 1 having images of 
the sea as the shooting subject. The cut 3 and the cut 4 
are collected as a scene 2 having images of the forest 
as the shooting subject. 

In the editing apparatus according to an embodi- 
es ment of the present invention, a representative frame 
included in frames forming each cut and a frame of a 
representative cut included in cuts forming each scene 
are displayed on the screen of the above described hier- 
archical structure as contracted pictures for high speed 
so display. 

In the embodiment of the present invention, editing 
can be simply conducted by altering the original hierar- 
chical structure displayed on the screen by simple 
manipulation. For example, it is possible to conduct 
55 alteration (editing) so as to generate such a hierarchical 
structure that the cut 1 and the cut 3 have been altered 
to a new scene 1 having animals inhabiting the island as 
the shooting subject. In the editing using the screen of 



SDOCID: <EP 08433 11 A2_l_> 



13 



EP0 843 311 A2 



14 



the hierarchical structure, it is possible to replace or 
erase cuts, or add different cuts. 

As for the method and system of moving image 
editing using the hierarchical structure, those disclosed 
in co-pending U.S. patent application Serial No. 
08/826.975 filed on April 9. 1997 titled "Method of Edit- 
ing Moving Image and Apparatus of Editing the Same" 
can be used in the embodiment of the present invention. 

For example, in the case where management infor- 
mation corresponding to the tree structure as shown in 
FIG. 10 has been read out, it is now assumed that a 
command for causing images of the cut 1 and cut 2 to 
compose the scene 1 is inputted by the operator. Then, 
image data having the data structure shown in FIG. 7 
and the tree structure management information having 
the data structure shown in FIG. 8, corresponding to the 
cut 1 and the cut 2 are read out from the magnetic stor- 
age device 506 by the CPU 501 . In addition, a picture 
number serving as "a parent" in the tree structure man- 
agement information is set to 1 , and values "1 " and "2" 
are set in addresses of cut numbers 1 and 2 serving as 
"children", respectively. 

Furthermore, display coordinates where icons of 
these two cuts are displayed are calculated. The calcu- 
lated values are set as cut icon display coordinates of 
FIG. 8, respectively. The tree structure management 
information is reregistered. 

As for a scene 2 as well, tree structure manage- 
ment information is reregistered by similar processing. 

In the case where the theme 1 generally formed by 
a plurality of scenes includes the scene 1 and the scene 
2 as shown in FIG. 10, the tree structure management 
information is read out from the magnetic storage 
device 506. Values "1 " and "2" are registered in the area 
of the CHILD SCENE NUMBERS 1 and 2. In the area of 
the PICTURE NUMBER of the management informa- 
tion having the scene number equal to the value "1" and 
the value "2," the value "theme V is respectively regis- 
tered. 

Subsequently, the operator manipulates the mouse 
(or keyboard) 509 and thereby issues a command that 
icons other than icons located in the lowest layer should 
not be displayed in the tree structure display window 
601. Upon this command, the hierarchical structure 
management information is read out from the magnetic 
storage device 506 or the memory 502. A display iden- 
tifier indicating that the icons should not be displayed in 
the tree structure display window 601 is added to prede- 
termined management information, and resultant infor- 
mation is reregistered. 

As a result, the CPU 501 discriminates this display 
identifier, recalculates the display position, and con- 
ducts the display on the monitor 508. As for the display 
at this time, therefore, the cuts 3 and 4 supplied with the 
display identifiers preventing the display are not dis- 
played as shown in FIG. 11. 

By successively repeating this manipulation, there- 
fore, the CPU 501 conducts processing on the basis of 



display coordinates of the icon. As shown in FIG. 12, the 
tree structure shown in FIG. 6 is thus displayed in the 
tree structure display window 601 of the monitor 508. 
Subsequently, transition to the effect image editing 

5 such as the wipe and dissolve is conducted. 

In the configuration of this embodiment, the screen 
display for effect image editing shown in FIG. 1 is first 
obtained by selecting an icon 610 or 611 representing 
the start of the wipe operation function out of icon fig- 

io ures arranged on the left side of FIG. 12. 

The tree structure shown in FIG. 6 has heretofore 
been described. In succession, therefore, description 
will be continued on this assumption. It is now assumed 
that the cut 1 and the cut 2 of FIG. 10 have been indi- 

75 cated by the operator as the subject of the effect editing. 
On the basis of coordinates of the two cuts indicated by 
the operator with the mouse 509 in the tree structure 
display window 601 shown in FIGS. 6 and 12, icon dis- 
play coordinates 620 stored in the hierarchical (tree) 

20 structure management information of FIG. 8 are 
referred to and information storing the corresponding 
cut numbers is respectively searched. 

Upon this indication manipulation, a window 101 
shown in FIG. 1 for creating an effect video picture is 

2$ opened on the screen of the monitor 508 (step 901 in a 
flow chart mentioned later). 

Subsequently, a contracted picture file name 622 
for high speed display stored in the second address 
from the address 621 of the information (Fig. 8) storing 

30 each of the searched cut numbers is read out. 

Furthermore, in accordance with the frame interval 
value for reading the contracted picture for high speed 
display prepared beforehand on the memory 502 (Fig. 
13), the contracted pictures for high speed display are 

35 read out from the magnetic storage device 506. The 
contracted pictures for high speed display of the cut 1 
and the cut 2 are displayed in the window 101 shown in 
FIG. 1 as icon streams (image streams) 102 and 103 
forming film images similar to framed photograph. 

40 it is now assumed that at this time "2" has been set 
by a slider 107 (Fig. 1) beforehand as the value specify- 
ing the frame interval. Then the frame numbers read out 
as the contracted pictures for high speed display 
become 1 , 3, 5, .... In other words, readout is performed 

45 every other frame. 

Since at this time the contracted pictures are used 
for the high speed display in this embodiment as 
described above, the processing time required for the 
display becomes shorter. 

so In a region located under the icon streams 102 and 
103, a film image synthesized from these two kinds film 
images by using the effect video processing is displayed 
as an icon train 104. 

In FIG. 1. the film image 104 synthesized by the 

55 scene transition of the dissolve effect is shown. 

In this embodiment therefore, the icon streams 1 02 
and 103 formed by film images similar to framed photo- 
graph are displayed as pseudo-pictures representing 
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the roll A and the roll B. As pseudo-pictures represent- 
ing the role A+B synthesized by the dissolve editing, the 
icon streams 104 formed by the film image similar to 
framed photograph is displayed. Thus the editing work 
of the role A+B can be conducted easily. $ 

Under the icon streams 102, 103 and 104 in the 
window 1 01 , a window 1 05 is set as an effect time set- 
ting area. In the window 105, rectangular figures (slid- 
ers) 150 and 151 selected and moved by the 
manipulation of the mouse 509 are displayed. The 10 
length of these rectangular figures 150 and 151 in the 
lateral direction on the screen represents the number of 
frames of the scene (cut 1 and cut 2 at this time). 

Therefore, the effect time setting can be grasped as 
overlap of these rectangular figures 150 and 151. By 75 
moving these figures 150 and 151 with the mouse 509, 
their lengths and overlap length are written into the 
memory 502. As a result, the effect time is set. 

A white area of a bar figure 1 06 in Fig. 1 indicates 
the overall length (time) of all images forming the cut 1 20 
and the cut 2 selected as the editing subject. A hatched 
rectangle 106a located at near the central part of the 
bar figure 106 indicates the time length of the con- 
tracted picture streams 1 02 and 1 03 of the cut 1 and cut 
2 of the currently displayed region and the temporal 25 
position of the displayed image in all of the images. 

By manipulating the mouse, the rectangle 106a can 
be moved. According to the moved position, film images 
of the place to be confirmed, of each scene of the effect 
portion are displayed as the icon streams 1 02, 103 and 30 
104. 

The position of the rectangle 106a moved by the 
mouse 509 at this time is also written into the memory 
502. As a result, the film images displayed on the icon 
trains 102, 103 and 104 are set 35 

As the setting area for displaying the picture data 
for high speed display on the film images with an arbi- 
trary reduced scale, i.e., as the setting area for display- 
ing the picture data for high speed display with a fixed 
frame interval, an icon of a film image icon stream dis- 40 
play interval setting slider 107 is set. In the slider 107 as 
well, a rectangle 107a moved by manipulating the 
mouse is displayed. At this time, the set numerical value 
is also displayed simultaneously within the rectangle 
1 07a as illustrated. 45 

In this embodiment, an OK button 108 and a cancel 
button 109 for icons are set. By manipulating a pointer 
on the screen with the mouse 509 and selecting the OK 
button 108, the operator can terminate the editing. By 
selecting the cancel button 109, the operator can aban- so 
don the processing on the way. 

Furthermore, in this embodiment, several kinds of 
wipe patterns and dissolve patterns, and an icon area 
1 10 for displaying other effect video processing by using 
figures are set. The operator selects a figure (icon) 55 
within this area 1 1 0 with the mouse 509. Thereby, a syn- 
thetic pattern of film images corresponding to the effect 
video processing indicated by the selected figure (icon) 



can be displayed. For example, a pattern 110a is an 
icon for a horizontal slide wipe effect wherein a picture 
of B role slides from the left side over a picture of A role 
thereby switching over the A role to B role. 

In this embodiment, therefore, various data 
required for effect video editing such as wipe and dis- 
solve are displayed as pictures. Furthermore, required 
input processing can be selected almost on the picture. 
Thus the work can be advanced efficiently. 

By referring to a flow chart, control processing of 
the window 101 conducted by the CPU 501 will now be 
described. 

The window 101 is controlled by using a predeter- 
mined program stored in the memory 502. FIG. 9 is a 
flow chart showing the control processing of the window 
101 conducted by this program. 

The processing of FIG. 9 is started by selecting 
"WIPE" 610 out of icon figures arranged on the left side 
of FIG. 1 2 with the mouse 509 as described above. First 
of all, initialization for making a variable 
"RED R AW_FLAG " . set in the memory 502 set equal to 
"0" is conducted at step 900. The variable 
"REDRAW_FLAG" is a flag for determining whether dis- 
play processing of the icon streams 102, 103 and 104 
should be conducted. 

Subsequently at step 901 , processing for displaying 
the window 101 (FIG. 1) on the monitor 508 is con- 
ducted. 

At this time, the window 101 shown in FIG. 1 is 
opened as to be displayed on the screen 600 of the 
monitor 508. 

Subsequently at step 902, interrupt input the 
processing is conducted, and the processing proceeds 
to steps 903. 

An interrupt caused by manipulating the mouse 509 
occurs when its button is turned OFF from ON as well 
as when its button is turned ON from OFF. At the step 
902, however, it is determined that an effective interrupt 
has been conducted when the button of the mouse 509 
is changed from OFF to ON. At this time, position coor- 
dinates of the pointer moved with the mouse 509 on the 
screen are inputted. 

At steps 903 through 908, the position coordinates 
of the pointer are compared with areas of various sliders 
and buttons and a wipe pattern mark of the window 101 
shown in FIG. 1 . Thereby, it is determined which event, 
i.e., which setting alteration and alteration quantity or 
value has been specified by the interrupt at this time. 

At this time, an interrupt conducted at a meaning- 
less place is disregarded. If an effective interrupt is not 
found, it is determined that there are no events and the 
processing proceeds to the step 909. 

If an event has occurred by pointing using the 
pointer and an interrupt using the button of the mouse 
509 at the steps 903 through 908, the processing pro- 
ceeds to execution of each of processing 91 1 through 
918 corresponding to that event. The processing 911 
through 918 will be described later. 
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After such processing has been finished, the step 
909 is executed. 

At the step 909 t it is determined whether the varia- 
ble REDRAW_FLAG is "1." Unless the variable 
REDRAW_FLAG is "1," the scene transition editing 
such as the wipe is interpreted as not commanded by 
the user and a return to the step 900 is conducted. 
Thereafter, the variable REDRAW_FLAG turning "1" is 
waited for in the loop of the steps 900 through 909. 

When the variable RED R AVV_FL AG is judged to be 
"1" at the step 909, the processing proceeds to step 
910. At step 910, data for special effect window setting 
are read from the memory 502, and processing for dis- 
playing the icon streams 102, 103 and 104 of the film 
images in the window 101 is conducted. Thereafter, the 
variable REDRAW_FLAG is reset and returned to 0, 
and the processing returns to the step 900. 

Therefore, processing of steps 900 through 910 is 
repeated until the end of the processing being able to be 
executed in accordance with displaying of window 101. 

Individual processing of steps 903 through 908 cor- 
responding to respective events will how be described. 

If a decision result at any one of steps 903 through 
906 has become "yes" as a result of selection with the 
pointer on the screen responsive to the mouse, then 
data for executing processing corresponding to the 
selected event are written into the memory 502 by the 
CPU 501 at steps 91 1 through 919. 

Various sliders and buttons displayed on the screen 
are associated with interrupt events beforehand. 

At step 903, the result becomes "yes" when the 
overlap quantity setting slider 105 has been pressed, 
i.e., the slider 105 has been pointed. Otherwise, the 
processing proceeds to the step 904. 

If the result is w yes," then the variable 
RED R AW_F LAG is set to "V at the step 91 1 and the 
processing proceeds to step 912. At the step 912, a 
numerical value inputted according to the point position 
specified by the mouse 509 is written in the address of 
the overlap quantity of the memory 502 t and the 
processing proceeds to step 909. 

At step 904, it is determined whether the display 
position setting slider 106 has been pressed. If the 
slider 106 has been pressed, the processing proceeds 
to the step 913. Otherwise, the processing proceeds to 
the step 905. 

At the step 91 3, the variable RED R AW_FLAG is set 
to "1" and the processing proceeds to step 914. At the 
step 914, a numerical value specified by the mouse 509 
is written into an address of a film image display start 
frame number of the memory 502 and the processing 
proceeds to the step 909. 

At step 905, it is determined whether the film image 
icon stream display interval setting slider 107 has been 
pressed, tf the slider 107 has been pressed, the 
processing proceeds to the step 915. Otherwise, the 
processing proceeds to the step 906. 

At the step 91 5, the variable R E D R A W_F LAG is set 



to "1" and the processing proceeds to step 916. At the 
step 916, a numerical value specified by the mouse 509 
is written into an address of a film image icon stream 
display interval number of the memory 502 and the 

5 processing proceeds to the step 909. 

At step 906, it is determined whether the wipe pat- 
tern mark 110 has been pressed. If the wipe pattern 
mark 110 has been pressed, the processing proceeds 
to the step 91 7. Otherwise, the processing proceeds to 

io the step 907. 

At the step 917, the variable RED R AW_F LAG is set 
to "1" and the processing proceeds to step 918. At the 
step 91 8, a numerical value specified by the mouse 509 
is written into an address of a wipe pattern number of 

is the memory 502 and the processing proceeds to the 
step 909. 

At step 907. it is determined whether the OK button 
1 08 has been pressed. If the OK button 1 08 has been 
pressed, the processing proceeds to the step 919. Oth- 
20 erwise, the processing proceeds to the step 908. 

At the step 91 9, the overlap quantity and the wipe 
pattern number read from a predetermined number of 
addresses beginning at the recording address of the 
pertinent cut number of the memory 502 are written into 
25 the hierarchical structure management information file 
of FIG. 8. and the processing proceeds to step 920. 

At step 920. processing for closing the window 101 
is conducted. 

Thereafter, a return to the original display screen 
30 shown in FIG. 12 is effected, and the effect editing is 
once finished. 

At step 908, it is determined whether the cancel 
button 109 has been pressed. If the cancel button 109 
has been pressed, the processing proceeds to the step 
35 920. Otherwise, the processing proceeds to the step 
909. 

At this time, therefore, the hierarchical structure 
management information file is not rewritten, and the 
processing is finished while maintaining the original 

40 information. 

The editing processing program represented by the 
flow chart shown in FIG. 9 is stored in the memory 502 
in a code signal form which can be read by the CPU 
501 . The memory 502 may be a recording medium such 

45 as a semiconductor memory, a CD-ROM, a magnetic 
disk or an optical disk. The editing processing program 
of the embodiment of the present invention represented 
by FIG. 9 may be installed in a user's general-purpose 
computer in such a form as to be stored in such a 

so recording medium. Alternatively, the editing processing 
program may be stored in an external storage device 
located in a remote place and downloaded to the mem- 
ory 502 via a communication line. 

In the embodiment of the present invention hereto- 

55 fore described, each scene to be edited and the picture 
subjected to the effect processing such as the wipe can 
be simultaneously displayed at the time of video editing 
using the scene transition such as the wipe. In the same 
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way as the case using the moving picture display, there- 
fore, it is possible to make the most of the advantages of 
the direct scheme and conduct various effect editing 
manipulations including the AB role editing efficiently 
and easily. 5 

Furthermore, in the embodiment of the present 
invention, it is not necessary to use hardware for 
processing such as the wipe or compression/expansion 
processing of pictures. Therefore, it is possible to suffi- 
ciently suppress the cost increase and the size increase 10 
of the apparatus. 

An embodiment in which sound information is 
edited by an image information editing system accord- 
ing to the present invention will now be described by 
referring to drawing. 15 

FIG. 13 shows an example of an apparatus used for 
such moving image editing. The same reference numer- 
als as those of the system in FIG. 5 basically refer to like 
components. 

A scene change point detector 503 analyzes the 20 
change of color distribution for each frame of moving 
pictures and detects a scene change point which is a 
change point between cuts. A picture contractor 
(reducer) 513 thins out pixels of an original image mate- 
rial and thereby creates moving pictures for high speed 25 
display. A magnetic storage device 506 stores the mov- 
ing pictures for high speed display and stores moving 
picture analyzing information describing the detection 
result supplied from the detector 503 and the moving 
pictures for high speed display. 30 

A monitor 508 displays the moving pictures stored 
in the magnetic storage device 506 on the basis of the 
analyzing information. The monitor 508 also provides 
the user with an editing screen for implementing a GUI 
(graphical user interface). By the GUI, the user can con- 35 
duct various data input manipulations including selec- 
tion out of predetermined display areas displayed in a 
picture conducted by manipulating an input device such 
as a mouse or a keyboard, and an output display corre- 
sponding to the manipulation is conducted. 40 

A frame buffer 507 stores images to be displayed 
on the monitor 508. A main memory 502 stores various 
control programs. A mouse or keyboard 509 is a com- 
mand input device of various operations. 

A video interface 504 is supplied with a video signal 45 
fed from a VTR (video tape recorder) 505, and converts 
the video signal to a moving image format handled by 
this apparatus. A picture compressor/expander 511 
conducts data compression on original video informa- 
tion and expands the compressed information. An audio so 
interface 512 conducts conversion processing on sound 
signals so that the sound signals may be exchanged 
between the VTR 505 and a computer. A CPU 501 con- 
trols the above described components and manages 
their operations. 55 

Together with the CPU 501 . the scene change point 
detector 503, the magnetic storage device 506, the 
monitor 508, the frame buffer memory 507, the main 



memory 502, the mouse or keyboard 509, the video 
interface 512, the picture compressor/expander 511, 
and the audio interface 512 are connected to a bus 510, 
and controlled by an access signal supplied from the 
CPU 501. 

Instead of the magnetic storage device 506, a dif- 
ferent storage medium such as an optical magnetic 
storage device may be used. Or a remote file connected 
to the bus 510 via a network may be used. 

On the basis of the scene change point detection 
information detected by the scene change point detec- 
tor 503, the CPU 501 takes out a picture located at a 
scene or cut change position out of the moving pictures 
for high speed display. The CPU 501 displays a repre- 
sentative picture of a scene or a cut to which that picture 
belongs on the monitor 508 as icon pictures 1302 
through 1304 in a tree structure display window 1301 
shown in FIG. 14. The term "icon" refers to a small pic- 
ture or figure for the screen display manipulation. 

The user (editing worker) handles pictures dis- 
played on the screen of the monitor 508 and shown in 
FIG. 14 as the subject of editing manipulation, and 
assembles them on the icon editing screen as the hier- 
archical structure (tree structure). Thus the user 
advances the editing. 

As for the method and system of the moving image 
editing using the hierarchical structure, those disclosed 
in co-pending U.S. patent application Serial No. 
08/826,975 can be used. 

In the tree structure of FIG. 14, a picture located in 
a middle class, for example, the picture 1 303 represents 
a plurality of pictures 1302 located in the lower class 
and belonging to the picture 1 303 as one group and cor- 
responds to a chapter or section of a story. Similarly, the 
picture 1304 located in the highest class represents, for 
example, a plurality of pictures 1303 located in the lower 
class and corresponds to the whole story. 

According to this method, editing can be conducted 
by taking a scene or a cut as the unit. Therefore, assem- 
bling of the story is facilitated. In addition, division of 
inputted moving images by taking a scene or a cut as 
the unit is conducted in a nearly automatic manner. As 
a result, the user is hardly troubled. In addition, rt is pos- 
sible to store and manage moving images in a hierarchi- 
cal structure. This results in an advantage that a 
required scene or cut can be searched for easily. The 
burden on the user in the moving image editing can thus 
be lightened. 

The editing system of the embodiment shown in 
FIG. 13 provides a moving image sound editing method 
facilitating the editing work of sounds accompanying the 
moving images, such as overlap editing or mixing edit- 
ing, by manipulating icons on the display screen. 

In the sound editing according to the embodiment 
of FIG. 1 3, pictures of a portion of image information to 
be edited are displayed as film images from the left to 
the right on the display screea By watching this film 
image display, it becomes possible to associate picture 
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information with sound information. Since the screen 
display manipulation with GUI can be conducted on cor- 
responding portions on lateral bar figures, editing of 
sound information is facilitated. 

Operation in an embodiment of the present inven- 5 
tion using the moving image editing apparatus of FIG. 
13 will now be described by referring to FIGS. 13 and 
15. In FIG. 15, an arrow represents control processing 
conducted by the CPU 501 . 

In response to a command fed from the operator via 10 
the mouse or keyboard 509, the picture signal and the 
sound signal are outputted from the video tape mounted 
on the VTR 505, and a process for registration of the 
video signal and the sound signal into the magnetic 
storage device 506 is started. At this time, the moving is 
image of one frame unit is supplied from the VTR 505 to 
the scene change point detector 503 via the video inter- 
face 504 and the bus 510 (processing 701 of FIG. 15). 

The scene change point detector 503 digitizes this 
inputted moving picture, analyzes the color distribution 20 
information between pictures, and detects a scene 
change point (processing 702). 

Moving image analyzing information (scene analyz- 
ing information) including the frame number of the 
scene change point generated by the scene change 25 
point detector 503 is stored in the magnetic storage 
device 506 (processing 703). 

At the same time, moving pictures for high speed 
display are stored in the magnetic storage device 506 
(processing 704). 30 

The moving pictures for high speed display are 
obtained by conducting the contraction processing 
(reduction processing) (processing 705) by which the 
imaginably same data as the moving images inputted to 
the scene change point detector 503 is contracted for 35 
conducting various displays according to the environ- 
mental size of the GUI. The moving pictures for high 
speed display become the subject of manipulation in the 
present editing apparatus. 

For returning an arbitrary image selected out of 40 
encoded moving images to a video signal and reproduc- 
ing the image, it is necessary to decode the encoded 
moving images. In the CPU 501 . the processing time for 
this decoding is required. 

In this embodiment, contracted pictures for high 45 
speed display are stored beforehand as described 
above. Therefore, decoding processing is not con- 
ducted, and decoding processing time for that purpose 
becomes unnecessary. The overall processing time can 
be thus made shorter. 50 

Simultaneously with the scene change point detec- 
tion processing (processing 702) and storage of the 
moving pictures for high speed display into the magnetic 
storage device 506 (processing 704), the moving pic- 
tures are encoded in the picture compressor 511 by ss 
using a picture compression technique (processing 
706) and the encoded moving pictures are stored in the 
magnetic storage device 506 (processing 707). 



The stored encoded moving pictures are used in 
the case where high speed display is not required, such 
as the case where the user desires to watch the final 
result of editing in detail. 

On the other hand, the sound output signal repro- 
duced from the VTR 505 also becomes digitized sound 
information via the audio interface 512 and is stored in 
the magnetic storage device 506 (processing 708). 

Then the editing worker (operator) commands rea- 
dout of the moving pictures for high speed display regis- 
tered in the magnetic storage device 506 by using the 
mouse or keyboard 509. 

In response to this, the CPU 501 reads out the 
scene analyzing information of the corresponding mov- 
ing image from the magnetic storage device 506. On the 
basis of that information, the CPU 501 reads out only 
the picture of the only first, for example, frame of each 
scene from the moving pictures for high speed display, 
and displays it in a main window 421 on the monitor 508 
as shown in FIG. 16. 

Icons such as moving picture display on the main 
window at this time become the subject of manipulation 
in this embodiment. 

The operator manipulates icons on the monitor dis- 
play screen, and assembles them into a hierarchical 
structure (tree structure) shown in the tree structure dis- 
play window 421 of FIG. 16. Thus the moving image 
according to the story is created. 

Hereafter, a concrete case will be described by tak- 
ing the overlap editing for the tree structure shown in 
FIG. 1 6 as an example. 

The case where overlap sound editing is conducted 
between a scene represented by an icon 422 (hereafter 
referred to as scene A) and a scene represented by an 
icon 425 (hereafter referred to as scene B) will now be 
considered. Transition to the editing mode is conducted 
by clicking the icon 422 and icon 425 with the mouse 
509 and thereafter clicking an overlap editing button 427 
included in a button group 426. As a result, an editing 
screen 420 is displayed. FIG. 1 7A is an enlarged view of 
this screen. 

In FIG. 17 A, numerals 121 and 122 denote con- 
tracted moving picture streams (image streams) 
obtained by arranging and displaying the high speed 
display moving pictures of respective frames in a prede- 
termined section of moving pictures (hereafter referred 
to as picture A and picture B) respectively belonging to 
the scene A and the scene B, from the left to the right in 
time series so as to form a film image. 

Numerals 123 and 124 denote waveform display 
portions of sounds corresponding to the scene A and 
the scene B (hereafter referred to as sound A and 
sound B). The waveforms are read out from the storage 
device 506. At this time, a splice position between 
scenes is denoted by 125. 

As for the display, the operator manipulates an icon 
128 for setting an interval between adjacent frames of a 
displayed picture of an image stream (hereafter referred 
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to as setting icon). As a result, contracted pictures are 
displayed every preset number of frames. Furthermore, 
by manipulating a display position setting icon 127, an 
arbitrary section in the whole of the image streams 121 
and 122 can be displayed. 5 

In order to conduct the overlap editing for adaptively 
controlling and varying the sound volume so as to atten- 
uate the sound A and augment the sound B in and near 
the splice position 125, the value of the sound volume 
control level is set according to each sound information, w 
The CPU 501 reads out sound information of editing 
subjects of the sound A and the sound B from the mag- 
netic storage device 506. On the basts of the sound 
information thus read out, the CPU 501 newly gener- 
ates sound information attenuated with predetermined 15 
attenuation factors in the editing subject portion. The 
CPU 501 rewrites the generated sound information into 
the magnetic storage device 506. 

As for the manipulation function in that case for 
making possible the sound volume control by selecting 20 
the range of the editing subject portion of each sound 
information and setting a predetermined attenuation 
factor of each sound information, it can be obtained by 
displayed figures including a lateral bar figure display 
portion 126. A lateral bar 126A corresponds to the 25 
sound A. The vertical width of its black portion repre- 
sents the volume control level set for the each predeter- 
mined period of the sound A. A lateral bar 126B 
corresponds to the sound B. The vertical width of its 
black portion represents the volume control level set for 30 
the each predetermined period of the sound B. 

According to the sound editing of the present inven- 
tion, not only the sound amplitude level but also a differ- 
ent sound parameter such as one of a plurality of sound 
quality levels or the musical interval can be adjusted in 35 
an arbitrarily specified editing section. For example, 
white circles 129 through 132 on the bar figure 126 are 
icons for specifying such adjustment sections of the 
sound parameter. The oblique slant portion of the bar 
figure 126 represents a gradual change of the sound 40 
parameter. Button icons 631 , 632 and 633 for specifying 
the sound amplitude level, sound quality level, and 
musical interval may be provided in the window as 
shown in FIG. 21. The operator selects a sound param- 
eter to be edited by clicking any one of the button icons 45 
631 , 632 and 633 with the mouse. If a button icon 634 is 
specified with the mouse, the sound in the editing sec- 
tion (at the time of fading) is subjected to an echo effect. 

The white circle figure 1 29 is a setting icon for set- 
ting a level attenuation start point of the sound A. The so 
white circle figure 130 is a setting icon for setting a level 
attenuation end point of the sound A. The white circle 
figure 131 is a setting icon for setting a level augmenta- 
tion start point of the sound B. The white circle figure 
1 32 is a setting icon for setting a level augmentation end 55 
point of the sound B. 

The image stream 121 and 122 of the film image, 
sound waveform display portions 123 and 124, the lat- 



eral bar display portion 126, the display position setting 
icon 127, and the display interval setting icon 128 are 
subjected to display control so as to be mutually linked. 
The result of manipulation is immediately reflected. 

While watching the image stream 121 and 122 
resembling film images and the sound waveforms, the 
operator conducts the overlap editing by manipulating 
the lateral bars 126A and 126B. 

Another editing screen 420 is shown in Fig. 17B. 

In FIG. 17B, numerals 121 and 122 denote con- 
tracted moving picture streams obtained by arranging 
and displaying the high speed display moving pictures 
of respective frames in a predetermined section of mov- 
ing pictures (hereafter referred to as picture A and pic- 
ture B) respectively belonging to the scene A and the 
scene B, from the left to the right in time series so as to 
form a film image. 

The portions of dotted line of the contracted moving 
pictures denote no efficient part for the edited picture. 

Numerals 123 and 124 denote waveform display 
portions of sounds corresponding to the scene A and 
the scene B (hereafter referred to as sound A and 
sound B). The waveforms are read out from the storage 
device 506. At this time, a splice position between 
scenes is denoted by 125. 

The explanation of the elements having the same 
numerals as FIG. 17A is omitted. Numerals 136 and 
137 denote edited picture image stream and sound 
image stream. 

By referring to FIG. 18, the sound editing process- 
ing using the lateral bars 126A and 126B in this embod- 
iment will now be described. By conducting a simple 
work of only specifying four white circle figures 129 
through 132 and moving them in the lateral direction 
while watching the video image, the operator can arbi- 
trarily set various parameters required for editing, such 
as a sound A attenuation start point position 521, a 
sound A attenuation end point position 522, a sound B 
augmentation start point position 523, a sound B aug- 
mentation end point position 524, split values 525 and 
526, fade time lengths 527 and 528, and a transition 
time length 529. 

For example, it is now assumed that the white circle 
figure 129 is subjected to mouse drag and thereby 
moved from the position 521 of (a) of FIG. 18 to a posi- 
tion 523 as shown in (b) of FIG. 18. In that case, the 
attenuation start point position 521 of the sound A 
becomes later by, for example, one frame of the images. 
Accordingly each of the split value 525 and the fade 
time length 527 of the sound A and the sound transition 
time length 529 between the sounds A and B is also 
reduced by. for example, a value corresponding to one 
frame. As a result, the sound A attenuates quicker than 
(a) of FIG. 18. 

It is now assumed that the white circle figure 130 is 
moved from the position 522 of (a) of FIG. 18 to a posi- 
tion 530 as shown in (c) of FIG. 18. In this case, the 
attenuation end of the sound A is made later by, for 
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example, one frame. Each of the fade time length of the 
sound A and the sound transition time length between A 
and B increases by, for example, a value corresponding 
to one frame. The sound A attenuates more gradually 
than the case of (a) of FIG. 18. 

On the other hand, it is now assumed that the white 
circle figure 131 is moved from the position 523 of (a) of 
FIG. 18 to the position 521 as shown in (d) of FIG. 18. In 
this case, the augmentation start time of the sound B is 
made earlier by, for example, one frame. Each of the 
split value and the fade time length of the sound B 
increases by, for example, a value corresponding to one 
frame. As a result the sound B augments gradually 
from an earlier time. 

It is now assumed that the white circle figure 132 is 
moved from the position 522 of (a) of FIG. 18 to a posi- 
tion 524 as shown in (e) of FIG. 18. In this case, the 
augmentation end time of the sound B is made earlier 
by. for example, one frame. Each of the fade length of 
the sound B and the sound transition time between A 
and B decreases by, for example, a value corresponding 
to one frame. The sound B augments faster. 

By doing so, therefore, in this embodiment, editing 
can be advanced easily so that the sound may be heard 
more smoothly even at the slice position between 
scenes. 

This embodiment is configured so that the move- 
ment of four white circle figures 129, 130, 131 and 132 
may be conducted also by clicking each picture of 
image streams 121 and 122 displayed on the screen 
420 by a film image, with the mouse. Furthermore, this 
embodiment is configured so that the position of the lat- 
eral bar display portion 126 itself may be dragged with 
the mouse/and association of the video pictures with 
sounds may be shifted. 

The embodiment is configured so that the result of 
editing may be determined at the end of the work by 
clicking an OK button 1 35. 

Thereby, moving pictures edited so as to incorpo- 
rate the result of manipulation are stored in the mag- 
netic storage device 506. At the same time, an alteration 
of the tree structure information is conducted. As a 
result, the display of the tree structure on the main win- 
dow of the moving image editing apparatus is changed 
as shown in FIG. 19. At this time, the information repre- 
senting that the editing including the sound has been 
carried out is also displayed on the main window of the 
moving image editing apparatus as a figure 1001 of 
FIG. 19. 

In the case where the operator wants to confirm the 
editing state before determining the editing, a PRE- 
VIEW button 133 of FIG. 17 may be clicked by the 
mouse. 

As a result, video pictures having a size of 640 by 
480 compressed by the picture compressor/ expander 
51 1 and stored are displayed in a window 430 of the 
monitor 508 as shown in FIG. 20. The editing state can 
thus be confirmed easily. 



This embodiment is configured so that editing may 
be conducted in the same way also in the case where 
special effect processing of video images such as the 
wipe or dissolve is desired to be conducted in a splice 

5 position between scenes. Hereafter, operation of sound 
editing accompanying the effect editing of video images 
will be described. 

A shift to the editing mode at this time is conducted 
by an operation similar to the above described overlap 

10 editing. In the main window shown in FIG. 16, the icons 
422 and 425 are clicked with the mouse 509. Thereaf- 
ter, a special effect processing button 428 or 429 such 
as the wipe or dissolve included in a button group 426 is 
clicked. As a result, the editing screen 420 after the shift 

15 becomes as shown in FIG. 21 . 

The display screen of FIG. 21 is an editing screen 
of the case where the wipe processing has been 
selected by the operator. Here, the image streams 121 
and 122, sound waveform display portions 123 and 124 

20 respectively of the sound A and the sound B, the lateral 
bar display portion 126, the lateral bars 126A and 126B, 
the display interval setting icon 128, the display position 
setting icon 127, and the white circle figures 129 
through 132 are the same as the foregoing description 

25 conducted with reference to FIG. 1 7A. 

In FIG. 21, numeral 605 denotes pictures after edit- 
ing obtained by conducting the wipe processing on the 
picture A and the picture B and displayed in a film image 
form as a contracted moving picture stream. The kind of 

30 the wipe effect at this time is determined by the opera- 
tor's selection for wipe patterns 606. 

Numeral 607 denotes a lateral bar display section 
representing the length of the wipe between the picture 
A and the picture B. Two lateral bars 607A and 607B 

35 represent the lengths of the portions in which the picture 
A and the picture B are displayed, respectively. There- 
fore, the overlapping portion of the lateral bars 607A 
and 607B represents the wipe length. 

The function of determining the degree of attenua- 

40 tion of the sound A and augmentation of the sound B in 
the overlap portion is implemented by the lateral bar dis- 
play portion 1 26. At this time, the white circle figure 1 29 
provides the attenuation start point of the sound A, and 
the white circle figure 130 provides the attenuation end 

45 point of the sound A. In the same way, the white circle 
figure 131 provides the augmentation start point of the 
sound B, and the white circle figure 132 provides the 
augmentation end point of the sound B. 

In these displays, the operator can select an arbi- 

so trary range for display by manipulating the display inter- 
val setting icon 128. By manipulating the display 
position setting icon 127. the operator can select an 
arbitrary section for display. At this time, the cut streams 
of the film image, the sound waveform display, the bars. 

55 and the icons are mutually linked. The result of manipu- 
lation for each bar or icon is immediately reflected to 
each other. 

The effect editing processing using the lateral bar 
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display portion 1 26 will now be described by referring to 
FIGS. 22 and 23. 

By manipulating the lateral bar display portion 607 
and the lateral bar display portion 126 while watching 
the moving image streams 121 and 122 each displayed s 
in a film image form and the sound waveform display 
portions 123 and 124 as shown in FIG. 21 , the operator 
conducts the overlap editing at the time of special effect 
processing of images such as the wipe and dissolve. 

First of all, (a^of FIG. 22 shows such a state that w 
the operator has provided the picture A and the picture 
B with an overlap portion by manipulating the lateral 
bars 607A and 607B with the mouse. The length of this 
overlap portion becomes the length of the wipe. 

At this time, the lateral bars 126A and 126B are 15 
linked to the manipulation conducted on the lateral bars 
607A and 607B, and automatically brought into the 
state shown in (a) of FIG. 22. 

It is now assumed in the state of (a) of FIG. 22 that 
a white circle figure 129 is dragged with the mouse to so 
move to a position shown in (b) of FIG. 22. Then a split 
value 715 of the sound A corresponding to, for example. 
1 frame is generated. Each of a fade time length 717 
and a sound transition time length 719 between A and B 
increases by a value corresponding to. for example, 1 25 
frame. Thus the attenuation start of the sound A can be 
made to begin earlier than the start of the wipe. 

On the other hand, if a white circle figure 130 is 
moved from the state of (a) of FIG. 22 to the position of 
(a) of FIG. 23. each of the fade time length 717 of the 30 
sound A and the sound transition time length 719 
between A and B increases by a value corresponding 
to, for example, 1 frame. Thus, it is possible to provide 
such an effect that the sound A is heard even after the 
wipe has been finished. 35 

If a white circle figure 131 is moved from the state 
of (a) of FIG. 22 to the position of (b) of FIG. 23, then a 
split value 716 of the sound B corresponding to, for 
example. 1 frame is generated, and a fade time length 
71 8 of the sound B decreases by a value corresponding 40 
to, for example, 1 frame. Thus the augmentation start of 
the sound B from the wipe start position is delayed. 

If a white circle figure 1 32 is moved from the state 
of (a) of FIG. 22 to the position of (c) of FIG. 23, then the 
fade time length 718 of the sound B decreases by a 45 
value corresponding to, for example, 1 frame and the 
augmentation of the sound B is completed earlier than 
the wipe termination time. 

In the special effect editing of images as well, there- 
fore, sounds can be heard more smoothly. so 

At this time as well, determination of the editing 
result and confirmation using the image reproduction 
before determination can be conducted in the same way 
as the above described overlap editing in a splice posi- 
tion between scenes. ss 

As a concrete example of the sound mixing editing 
operation according to this embodiment, the case of 
editing where a BGM (background music) is mixed with 
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the scene represented by the icon 1303 displayed by 
the tree structure shown in FIG. 14 will now be 
described. 

First of all, manipulation preceding the shift to the 
sound mixing editing mode will now be described. 

In this case, a first requirement is that sound data 
containing the BGM is stored in the magnetic storage 
device 506 beforehand. 

On this premise, a command for reading out this 
sound data into the present editing apparatus is given 
via the mouse or keyboard 509. 

Thereupon, the CPU 501 creates an icon 1305 
(hereafter referred to as sound icon) displayed as same 
as the icons 1302 through 1304 of the moving pictures 
shown in FIG. 14. 

The sound icon 1 305 required to handle sound data 
on the editing apparatus is thus set. Thereafter, by click- 
ing this sound icon 1305 with the mouse in the same 
way as the icon of the moving pictures, it becomes pos- 
sible to conduct sound reproduction and insertion work. 

The method for shifting to the sound mixing editing 
mode will now be described. 

There are the following two ways of shifting to the 
mode. In a first method, the sound icon 1305 is dragged- 
with the mouse and dropped at the icon 1303. In a sec- 
ond method, the sound icon 1305 and the icon 1303 are 
selected, and a predetermined button of a button group 
426 is clicked with the mouse. As a result, the screen 
display is changed, and the editing screen after the shift 
becomes as shown in FIG. 24. 

In FIG. 24, numeral 931 denotes a contracted mov- 
ing picture stream obtained by displaying frames of a 
fixed section of moving images belonging to a scene ^ 
represented by the icon 1303 (Fig. 14), in a film image 
form in the same way as the above described case. At 
this time, the sound belonging to the scene represented 
by the icon 1303 is displayed in a waveform display por- 
tion 932, and the waveform of the sound to be mixed is 
displayed in a waveform display portion 933. 

The vertical width of a black lateral bar 934M dis- 
played in a lateral bar display portion 934 represents the 
volume control level set for the each predetermined 
period of the sound to be mixed. A white circle figure 
935 represents a mixing augmentation completion 
point. 

Here, each of the display position setting icon 127, 
the display interval setting icon 128, the OK button 135, 
and the PREVIEW button 133 has the same function as 
that in the case described before. Those figures, the film 
image, and the sound waveform display are mutually 
linked. Thus, results of manipulation for respective bars 
and icons are immediately reflected to each other. 

With reference to the mixing editing screen of FIG. 
24, the operator can first set a sound mixing augmenta- 
tion completion point by specifying the white circle fig- 
ure 935 with the mouse and moving the figure on the 
lateral bar 934M while watching the moving picture 
stream 931 displayed in the film image. 
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At this time, the lateral bar 934M has a function of 
gradually augmenting the mixing sound at the time of 
mixing start as illustrated so that sounds may be mixed 
without unnaturalness. Although not illustrated, the lat- 
eral bar 934M has also a function of gradually attenuat- 
ing the mixing sound at the time of mixing end. 

In this embodiment, it is possible to provide the 
sound mixing augmentation completion point even by 
clicking the moving picture stream 931 displayed in the 
film image with the mouse. 

If in the state shown in (a) of FIG. 25 it is desired to 
make a frame 931 F included in the picture stream 931 
the mixing augmentation start point of the sound infor- 
mation, this frame 93 1F is clicked with the mouse. In 
linkage thereto, the figure 935 on the lateral bar 934M 
representing the level of the sound to be mixed is moved 
to the frame 931 F as shown in (b) of FIG. 25. 

Then a white circle figure 935 is moved with the 
mouse as shown in (c) of FIG. 25. Thereupon, mixing is 
conducted so that the sound volume control level may 
be gradually increased in a section 936. 

Although not illustrated, the end time point of the 
sound mixing can also be specified on the lateral bar 
934M of the sound with the mouse. 

In the case where the editing processing is finished 
and the editing result is to be determined, the OK button 
135 (FIG. 14) is clicked with the mouse. 

Moving images with the results of the editing 
processing heretofore described reflected therein are 
stored in the magnetic storage device 506. At the same 
time, alteration of the tree structure information is con- 
ducted. As a result, the display returns to the original 
main window. At this time, the display of the tree struc- 
ture becomes as shown in FIG. 26. In the same way as 
the above described case, the information that the mix- 
ing editing of the sound information 1305 has been car- 
ried out is displayed on the main window 1301 as a 
figure 1101. 

On the other hand, if it is desired to confirm the edit- 
ing state before the completion of the editing work, the 
PREVIEW button is clicked with the mouse. Thereupon, 
the video pictures having a size of 640 by 480 com- 
pressed by the picture compressor/expander 51 1 and 
stored are reproduced on the monitor 508. In the same 
way as the case described before, the editing state can 
be thus confirmed easily. 

According to the present embodiment the relation 
between the picture and the sound is obtained so as to 
be quite obvious. While watching the video pictures, 
therefore, it is possible to easily determine, amend and 
alter the sound editing point. 

Furthermore, according to the present invention, 
pictures used for editing are obtained as intuitive dis- 
plays akin to human sensitivity. In sound editing, there- 
fore, skill is not especially required. In this point as well, 
the editing is facilitated. 

By referring to flow charts of FIGS. 27, 28 and 29, 
procedures of the case where the editing processing 



using the editing screens shown in FIGS. 17A, 21 and 
24 is conducted under the control of a computer will now 
be described. 

FIG. 27 is a flow chart showing the procedure of the 

5 overlap editing shown in FIG. 1 7A. If the overlap editing 
mode is selected by the operator, " R E D R AW_F LAG" 
set in the memory 502 is initialized to a value of "0" at 
step 140. At step 141, the window 420 shown in FIG. 
1 7A is displayed on the monitor 508. At step 142, inter- 

10 rupt input processing from the mouse 509 is conducted. 
At step 143, it is determined whether an event of 
pressing the setting icon 127 has occurred. If the dis- 
play position setting icon 127 has been pressed, the 
"REDRAW_FLAG M is set to a value of "1" at step 144. 

is Furthermore, a display position specified by the opera- 
tor is written into the memory 502 at step 1 45. 

If it is determined at the step 143 that there is no 
occurrence of an event, it is determined at step 146 
whether an event of pressing the display interval setting 

20 icon 128 has occurred. If the display position setting 
icon 128 has been pressed, the t, REDRAW_FLAG" is 
set to a value of "I" at step 147. Furthermore, a display 
interval specified by the operator is written into the 
memory 502 at step 148. 

25 If it is determined at the step 146 that there is no 
occurrence of an event, it is determined at step 149 
whether an event of pressing any one of sound param- 
eter icons (buttons) 631 through 633 has occurred. If 
any one of sound parameter icons 631 through 633 has 

30 been pressed, the "REDRAW_FLAG ,t is set to a value of 
"1" at step 150. Furthermore, a kind of a control param- 
eter specified by the operator is written into the memory 
502 at step 151. 

If it is determined at the step 149 that there is no 

35 occurrence of an event, it is determined at step 152 
whether an event of pressing an echo icon (button) 634 
has occurred. If the echo icon 634 has been pressed, 
the "REDRAW_FLAG n is set to a value of "1" at step 
153. Furthermore, information of the echo effect is wrtt- 

40 ten into the memory 502 at step 154. 

If it is determined at the step 152 that there is no 
occurrence of an event, it is determined at step 155 
whether an event of pressing the white circle figure 129 
has occurred. If the white circle figure 129 has been 

45 pressed, the "REDRAW_FLAG" is set to a value of M V 
at step 156. Furthermore, the fade out start time of the 
sound A is written into the memory 502 at step 1 57. 

If it is determined at the step 1 55 that there is no 
occurrence of an event, it is determined at step 158 

so whether an event of pressing the white circle figure 130 
has occurred. If the white circle figure 130 has been 
pressed, the "REDRAW_FLAG W is set to a value of H r 
at step 159. Furthermore, the fade out end lime of the 
sound A is written into the memory 502 at step 160. 

55 If it is determined at the step 158 that there is no 
occurrence of an event, it is determined at step 161 
whether an event of pressing the white circle figure 131 
has occurred. If the white circle figure 131 has been 
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pressed, the "REDRAV^FLAG" is set to a value of "1" 
at step 162. Furthermore, the fade in start time of the 
sound B is written into the memory 502 at step 163. 

If it is determined at the step 161 that there is no 
occurrence of an event, it is determined at step 164 
whether an event of pressing the white circle figure 132 
has occurred. If the white circle figure 132 has been 
pressed, the "REDRAW_FLAG" is set to a value of "1" 
at step 165. Furthermore, the fade in end time of the 
sound B is written into the memory 502 at step 166. 

If it is determined at the step 164 that there is no 
occurrence of an event, it is determined at step 167 
whether an event of pressing the picture stream 121 has 
occurred. If the picture stream 121 has been pressed, 
the "REDRAW_FLAG K is set to a value of "1" at step 
168. Furthermore, the fade in/out time information of 
each of the sound A and the sound B is written into the 
memory 502 according to the preselected state of each 
white circle figure at step 169. 

If it is determined at the step 167 that there is no 
occurrence of an event, it is determined at step 170 
whether an event of pressing the picture stream 122 has 
occurred. If the picture stream 122 has been pressed, 
the "REDRAW_FLAG" is set to a value of "1" at step 
171. Furthermore, the fade in/out time information of 
each of the sound A and the sound B is written into the 
memory 502 according to the preselected state of each 
white circle figure at step 172. 

If it is determined at the step 170 that there is no 
occurrence of an event, it is determined at step 173 
whether an event of pressing the lateral bar display por- 
tion 126 has occurred. If the lateral bar display portion 
126 has been pressed, the "REDRAWIFLAG" is set to 
a value of "1" at step 174. Furthermore, the value of 
association relation between the video picture and the 
sound is written into the memory 502 at step 1 75. 

If it is determined at the step 173 that there is no 
occurrence of an event, it is determined at step 176 
whether an event of pressing the PREVIEW button 133 
has occurred, if the PREVIEW button 133 has been 
pressed, the preview window 430 is opened on the dis- 
play screen at step 1 77. Furthermore, the window 420 is 
closed on the display screen at step 1 78. 

If it is determined at the step 176 that there is no 
occurrence of an event, it is determined at step 179 
whether an event of pressing the OK button 135 has 
occurred. If the OK button 135 has been pressed, then 
the picture (image) of the editing result is displayed on 
the monitor and the editing result is stored in the mag- 
netic storage device 506 at step 180. 

If it is determined at the step 179 that there is no 
occurrence of an event or after the processing of the 
step 180 is conducted, it is determined at step 181 
whether an event of pressing the CANCEL button 134 
has occurred. If the CANCEL button 134 has been 
pressed, the processing of the step 178 is conducted. 

If it is determined at the step 181 that there is no 
occurrence of an event or after the processing of one of 



the steps 145, 148. 151, 154. 157, 160, 163, 166, 169, 
172, and 175 is conducted, it is determined at step 182 
whether the "REDRAW_FLAG" is H 1." If it is determined 
at the step 182 that the "RED R AW_FLAG" is "I," the 

5 picture corresponding to the edited contents is dis- 
played on the monitor, at step 183. Furthermore, the 
processing returns to the step 140. If it is determined at 
the step 182 that the "REDRAW_FLAG" is not "1." the 
processing returns to the step 140. 

10 If the processing of the step 178 is conducted, the 
overlap editing mode is once finished. 

Next, FIG. 28 is a flow chart showing the procedure 
of the wipe editing shown in FIG. 21. If the wipe editing 
mode is selected by the operator, "REDRAW_FLAG" 

15 set in the memory 502 is initialized to a value of "0" at 
step 240. At step 241 , the window 420 shown in FIG. 21 
is displayed on the monitor 508. At step 242, interrupt 
input processing from the mouse 509 is conducted. 
At step 243, it is determined whether an event of 

20 pressing the setting icon 127 has occurred, rf the dis- 
play position setting icon 127 has been pressed, the 
"REDRAW^FLAG" is set to a value of "1" at step 244. 
Furthermore, a display position specified by the opera- 
tor is written into the memory 502 at step 245. 

25 If it is determined at the step 243 that there is no- 
occurrence of an event, it is determined at step 246 
whether an event of pressing the display interval setting 
icon 128 has occurred. If the display position setting 
icon 128 has been pressed, the "RED RAW_FLAG" is 

30 set to a value of "1 " at step 247. Furthermore, a display 
interval specified by the operator is written into the 
memory 502 at step 248. 

H it is determined at the step 246 that there is no, 
occurrence of an event, it is determined at step 249 

35 whethier an event of pressing the lateral bar 607 has 
occurred. If the lateral bar 607 has been pressed, the 
"REDRAW_FLAG M is set to a value of "1" at step 250. 
Furthermore, an overlap quantity of the video pictures 
specified by the operator is written into the memory 502 

40 at step 251. 

If it is determined at the step 249 that there is no 
occurrence of an event, it is determined at step 252 
whether an event of pressing the white circle figure 129 
has occurred. If the white circle figure 129 has been 

45 pressed, the "REDRAW_FLAG n is set to a value of "1" 
at step 253. Furthermore, the fade out start time of the 
sound A is written into the memory 502 at step 254. 

If it is determined at the step 252 that there is no 
occurrence of an event, it is determined at step 255 

so whether an event of pressing the white circle figure 130 
has occurred. If the white circle figure 130 has been 
pressed, the "REDRAW_FLAG H is set to a value of "1" 
at step 256. Furthermore, the fade out end time of the 
sound A is written into the memory 502 at step 257. 

55 If it is determined at the step 255 that there is no 
occurrence of an event, it is determined at step 258 
whether an event of pressing the white circle figure 131 
has occurred. If the white circle figure 131 has been 



17 

ISDOCID: <EP 084331 1A2_I_> 



33 



EP 0 843 311 A2 34 



pressed, the "REDRAW_FLAG" is set to a value of "1" 
at step 259. Furthermore, the fade in start time of the 
sound B is written into the memory 502 at step 260. 

If it is determined at the step 258 that there is no 
occurrence of an event, it is determined at step 261 
whether an event of pressing the white circle figure 132 
has occurred. If the white circle figure 132 has been 
pressed, the "REDRAW_FLAG" is set to a value of "1" 
at step 262. Furthermore, the fade in end time of the 
sound B is written into the memory 502 at step 263. 

If it is determined at the step 261 that there is no 
occurrence of an event, it is determined at step 264 
whether an event of pressing the picture stream 121 has 
occurred. If the picture stream 121 has been pressed, 
the "REDRAW_FLAG" is set to a value of "1" at step 
265. Furthermore, the fade in/out time information of 
each of the sound A and the sound B is written into the 
memory 502 according to the preselected state of each 
white circle figure at step 266. 

If it is determined at the step 264 that there is no 
occurrence of an event, it is determined at step 267 
whether an event of pressing the picture stream 122 has 
occurred. If the picture stream 122 has been pressed, 
the "RED RAW__FLAG" is set to a value of T at step 
268. Furthermore, the fade in/out time information of 
each of the sound A and the sound B is written into the 
memory 502 according to the preselected state of each 
white circle figure at step 269. 

If it is determined at the step 267 that there is no 
occurrence of an event, rt is determined at step 270 
whether an event of pressing the lateral bar display por- 
tion 126 has occurred, tf the lateral bar display portion 
126 has been pressed, the "RED R AW_FLAG" is set to 
a value of "1" at step 271. Furthermore, the value of 
association relation between the video picture and the 
sound is written into the memory 502 at step 272. 

If it is determined at the step 270 that there is no 
occurrence of an event, it is determined at step 273 
whether an event of pressing the PREVIEW button 133 
has occurred. If the PREVIEW button 133 has been 
pressed, the preview window 430 is opened on the dis- 
play screen at step 274. Furthermore, the window 420 is 
closed on the display screen at step 275. 

If it is determined at the step 273 that there is no 
occurrence of an event, it is determined at step 276 
whether an event of pressing the OK button 135 has 
occurred, if the OK button 135 has been pressed, the 
picture (image) of the editing result is displayed on the 
monitor and the editing result is stored in the magnetic 
storage device 506 at step 277. 

If it is determined at the step 276 that there is no 
occurrence of an event or after the processing of the 
step 275 is conducted, it is determined at step 278 
whether an event of pressing the CANCEL button 134 
has occurred. If the CANCEL button 134 has been 
pressed, the processing of the step 275 is conducted. 

If it is determined at the step 278 that there is no 
occurrence of an event or after the processing of one of 



the steps 245, 248, 251, 254, 257. 260, 263, 266, 269, 
and 272 is conducted, it is determined at step 279 
whether the "REDRAW^FLAG" is "1 If it is determined 
at the step 279 that the "REDRAW_FLAG H is "V the 

5 picture corresponding to the edited contents is dis- 
played on the monitor, at step 280. Furthermore, the 
processing returns to the step 240. If it is determined at 
the step 279 that the "REDRAW„FLAG M is not-"V the 
processing returns to the step 240. 

io If the processing of the step 275 is completed, the 
wipe editing mode is once finished. 

Next, FIG. 29 is a f tow chart showing the procedure 
of the mixing editing shown in FIG. 24. If the mixing edit- 
ing mode is selected by the operator, 

75 "REDRAW_FLAG" set in the memory 502 is initialized 
to a value of "0" at step 340. At step 341, the window 
420 shown in FIG. 24 is displayed on the monitor 508. 
At step 342, interrupt input processing from the mouse 
509 is conducted. 

20 At step 343. it is determined whether an event of 
pressing the display position setting icon 127 has 
occurred. If the display position setting icon 127 has 
been pressed, the "RED R AW_FLAG" is set to a value of 
"1" at step 344. Furthermore, a display position speci- 

25 f ied by the operator is written into the memory 502 at 
step 345. 

If it is determined at the step 343 that there is no 
occurrence of an event, it is determined at step 346 
whether an event of pressing the display interval setting 

30 icon 128 has occurred. If the display position setting 
icon 128 has been pressed, the "RED R AW_FLAG" is 
set to a value of "1" at step 347. Furthermore, a display 
interval specified by the operator is written into the 
memory 502 at step 348. 

36 If it is determined at the step 346 that there is no 
occurrence of an event, it is determined at step 349 
whether an event of pressing the lateral bar display por- 
tion 934 has occurred. If the lateral bar display portion 
934 has been pressed, the "REDRAW_FLAG" is set to 

40 a value of "1 " at step 350. Furthermore, the fade in start 
time of the mixing sound specified by the operator is 
written into the memory 502 at step 351 . 

tf it is determined at the step 349 that there is no 
occurrence of an event, it is determined at step 352 

45 whether an event of pressing the picture stream 931 has 
occurred. If the picture stream 931 has been pressed, 
the "RED R AW_FLAG" is set to a value of "1" at step 
353. Furthermore, the fade in start time of the sound is 
written into the memory 502 at step 354. 

so If it is determined at the step 352 that there is no 
occurrence of an event, it is determined at step 355 
whether an event of pressing the white circle figure 935 
has occurred. If the white circle figure 935 has been 
pressed, the * , REDRAW_FLAG" is set to a value of "1" 

55 at step 356. Furthermore, the fade in end time of the 
sound is written into the memory 502 at step 357. 

If it is determined at the step 355 that there is no 
occurrence of an event, it is determined at step 358 
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whether an event of pressing the PREVIEW button 133 
has occurred. If PREVIEW button 133 has been 
pressed, the preview window 430 is opened on the dis- 
play screen at step 359. Furthermore, the window 420 is 
closed on the display screen at step 360. 5 

If it is determined at the step 358 that there is no 
occurrence of an event, it is determined at step 361 
whether an event of pressing the OK button 135 has 
occurred. If the OK button 135 has been pressed, the 
picture (image) of the editing result is displayed on the 10 
monitor and the editing result is stored in the magnetic 
storage device 506 at step 362. 

If it is determined at the step 361 that there is no 
occurrence of an event or after the processing of the 
step 362 is conducted, it is determined at step 363 15 
whether an event of pressing the CANCEL button 134 
has occurred. If the CANCEL button 134 has been 
pressed, the processing of the step 360 is conducted. 

If rt is determined at the step 363 that there is no 
occurrence of an event or after the processing of one of 20 
the steps 345, 348, 351 , 354, and 357 is conducted, it is 
determined at step 364 whether the "RED R A W_F LAG" 
is "1." If it is determined at the step 364 that the 
"RED R AW_FLAG" is "1," the picture corresponding to 
the edited contents is displayed on the monitor at step 25 
365. Furthermore, the processing returns to the step 
340. If it is determined at the step 364 that the 
"REDRAW_FLAG" is not "1." the processing returns to 
the step 340. If the processing of the step 360 is con- 
ducted, the overlap editing mode is once finished. 30 

The editing processing programs represented by 
the flow charts shown in FIGS. 27 through 29 are stored 
on the memory 502 in a code signal form which can be 
read by the CPU 501. The memory 502 may be a 
recording medium such as a semiconductor memory, a 35 
CD-ROM, a magnetic disk, or an optical disk: 

It is evident that the present invention is not limited 
to the embodiments heretofore described, and persons 
skilled in the art may conduct various improvements 
and modifications on the basis of the disclosure. 40 

Claims 

1. An image information material editing method for 
conducting editing processing on an image infor- 45 
mation material with an aid of computer, comprising 
the steps of: 

storing information of said image information 
material into a storage device; so 
reading out a plurality of image streams to be 
edited within said image information material 
from said storage device; 
editing said plurality of image streams so as to 
generate an edited image stream; and ss 
simultaneously displaying the image streams 
to be edited and said edited image streams. 



2. An image information material editing method 
according to claim 1 , wherein said displaying step 
comprises a step of displaying information repre- 
sentative of editing performed on said image 
streams to be edited. 

3. An image information material editing method 
according to claim 2, wherein said information rep- 
resents the position of the editing in said image 
streams to be edited. 

4. An image information material editing method 
according to claim 1 , wherein said image streams 
read out from said storage device is displayed on 
the screen in first and second film images (102, 
103) formed by a connection of a plurality of frame 
images; 

a kind of effect editing and an editing range to 
be conducted on said image streams is speci- 
fied; and 

the image streams with the specified effect 
editing processing conducted thereon is dis- 
played on the screen as a third film image (1 04) 
together with said first and second film images 
(102, 103). 

5. An image information material editing method 
according to claim 4, further comprising the step/of 
creating (513) contracted images by reducing pixels 
of said image streams and storing the contracted 
image streams in said storage device (506), 

wherein the image streams of said film 
image are created from said contracted images- 
read out from said storage device and displayed. 

6. An image information material editing method 
according to claim 5, further comprising the step of 
representing a specified editing range of said image 
streams by bar figures (105, 106) each having a 
time axis as a longitudinal direction thereof, and 
displaying said bar figures on the screen together 
with said first to third film images. 

7. An image information material editing method 
according to claim 6, further comprising the step of 
altering the editing range by altering the lengths of 
said bar figures. 

8. An image information material editing method 
according to claim 7, wherein in case of editing of 
mutually overlapping and coupling portions of first 
and second moving picture information, 

a first bar figure (102) corresponding to said 
first moving picture information and a second 
bar figure (103) corresponding to said second 
moving picture information are displayed on the 
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screen; 

said first and second bar figures are displayed 
so that an overlap region of the first and second 
moving picture information may be recognized; 
and 5 
pictures obtained by mutually overlapping and 
coupling the first and second moving picture 
information in said overlap region are displayed 
on said screen as said third film image (104). 

10 

9. An image information material editing method 
according to claim 8, wherein said editing by mutu- 
ally overlapping and coupling portions of the first 
and second moving picture information is editing 
processing of gradually changing from pictures of is 
the first moving picture information to pictures of 
said second moving picture information in said 
overlap region. 

10. An image information material editing method 20 
according to claim 1 , wherein said moving picture 
information and sound information (123, 124, 126) 
relating to said moving picture information are dis- 
played on said screen in a common time axis; 

25 

editing processing (631-634) for adjusting 
parameters relating to sounds of the sound 
information in said editing position is com- 
manded by the user; and 

a result (126A, 126B) of editing of the sound 30 
information commanded according to said 
command is displayed on said screen. 

11. An image information material editing method 
according to claim 10, wherein in case of editing of 35 
displaying said moving picture information in a film 
image, displaying film images (121 , 122) of a plural- 
ity of different moving picture information pieces on 

the screen so as to be distinguishable from each 
other, and mutually coupling said different moving 40 
picture information pieces: 

editing processing of adjusting the parameters 
relating to the sounds of the sound information 
in said editing position is arbitrarily com- 45 
manded by the user; and 
the result of editing of the sound information 
commanded according to said command is dis- 
played on said screen. 

50 

12. An image information material editing method 
according to claim 1 1 , wherein in case of overlap 
editing of mutually overlapping portions of said dif- 
ferent moving picture information pieces: 

55 

an overlap region of said different moving pic- 
ture information pieces is displayed on the 
screen; 



editing processing of adjusting the parameters 
relating to the sounds of the sound information 
in said overlap region is arbitrarily commanded 
by the user; and 

the result of editing of the sound information 
commanded according to said command is dis- 
played on said screen. 

13. An image information material editing method 
according to claim 12, wherein a displayed editing 
position of said sound information is changed so as 
to be linked to an alteration of the editing position of 
said moving picture information. 

14. An image information material editing method 
according to daim 12, wherein editing processing 
for changing an amplitude level of the sound of the 
sound information in said editing position is com- 
manded by the user; and 

a change state of a sound level in the editing 
position specified according to said command 
is displayed on said screen. 

15. An image information material editing method 
according to claim 14. wherein the amplitude level 
of the sound is displayed on the screen by using a 
bar figure; and 

a change point of the amplitude level is dis- 
played on said bar figure by using a specific fig- 
ure. 

16. An image information material editing method 
according to claim 15, wherein a change start point 
(129, 131) and a change end point (130, 132) of 
said amplitude level are displayed on said bar figure 
by using said specific figure. 

17. An image information material editing method 
according to claim 10, wherein in case of editing of 
displaying said moving picture information in a film 
image, and mixing predetermined sound informa- 
tion in a portion of said moving picture information: 

a position (935, 935M) of the moving picture 
information in which said predetermined sound 
information is to be inserted is arbitrarily speci- 
fied by the user's command; 
said predetermined sound information is cou- 
pled with said moving picture information in the 
specified position in accordance with sad com- 
mand; and 

a figure (1101) indicating that said predeter- 
mined sound information is mixed with said 
moving picture information is displayed on said 
screen. 
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18. An image information material editing method 
according to claim 1 7, wherein when said predeter- 
mined sound information is stored in said storage 
device, a figure (1103) identifying said predeter- 
mined sound information is displayed on said 5 
screen; and 

when the user specifies said figure identifying 
said predetermined sound information and 
superimposes said figure on a film image of 10 
desired moving picture information, said prede- 
termined sound information is mixed with the 
moving picture information with said figure 
identifying said predetermined sound informa- 
tion superimposed thereon. 15 

19. An image information material editing method 
according to claim 10, further comprising the step 
of creating contracted moving pictures by thinning 
pixels of said moving picture information and stor- 20 
ing the contracted moving pictures in said storage 
device, 

wherein the moving pictures of said film 
image are created from said contracted moving pic- 
tures read out from said storage device and dis- 25 
played. 

20. An image information material editing system for 
conducting editing processing on a video informa- 
tion material containing sound information and 30 
moving picture information with the aid of a compu- 
ter (501), said video information material editing 
system comprising: 

a storage device (506) for storing information of 35 
said video information material; 
means (501) for reading out said video infor- 
mation from said storage device; 
a display device (508) for displaying said read 
out video information; and 40 
editing control means (501) responsive to a 
user's command, for displaying an editing posi- 
tion of said video information, altering said edit- 
ing position, conducting commanded editing 
processing on video information, and display- 45 
ing edited video information on a screen of said 
display device. 

21. An image information material editing system 
according to claim 20, wherein said editing control so 
means (50 1 ) comprises: 

means (101. 103. 121, 122) for forming first 
and second film images formed by a connec- 
tion of a plurality of frame pictures from said 55 
moving picture information read out from said 
storage device; 

means (104, 105) for forming a third film image 



from the moving picture information with the 
specified picture editing processing conducted 
thereon; and 

means for displaying said first to third film 
images together on the screen. 

22. An image information material editing system 
according to claim 21 , further comprising: 

picture contracting means for creating con- 
tracted moving pictures by thinning out pixels of 
said moving picture information and storing the 
contracted moving pictures in said storage 
device; and 

means for creating the moving pictures of said 
film image from said contracted moving pic- 
tures read out from said storage device. 

23. An image video information material editing system 
according to claim 20, wherein said said editing 
control means comprises: 

means for displaying said moving picture infor- 
mation and sound information relating to said 
moving picture information on said screen in a 
common time axis; and 
means for displaying a result of editing of the 
sound information commanded according to a 
command for adjusting parameters relating to 
sounds of the sound information in said editing 
position. 

24. A computer program product comprising a compu- 
ter usable medium having computer readable pro- 
gram code means embodied in said medium for 
conducting editing processing of a video informa- 
tion material containing sound information and 
moving picture information, said computer readable 
program code means comprising: 

means for reading out said video information 
from a storage device storing information of 
said video information material; 
• means for displaying said read out video infor- 

mation on a display device; 
means for displaying an editing position of said 
video information on said display device 
according to a command from a user; 
means for altering said editing position upon a 
command from the user; 
means for conducting editing processing spec- 
ified by the user on video information located in 
the displayed editing position; and 
means for displaying edited video information 
on the screen. 

25. A computer program product according to claim 24, 
said computer readable program code means fur- 
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ther comprising: 

means for displaying said moving picture infor- 
mation read out from said storage device on 
the display device in first and second film 5 
images formed by a connection of a plurality of 
frame pictures; and 

means responsive to a kind of picture editing 
and an editing range to be conducted on said 
moving picture information, for displaying mov- 10 
ing picture information with the specified pic- 
ture editing processing conducted thereon on 
said display device in a third film image 
together with said first and second film images. 

75 

26. A computer program product according to claim 24, 
said computer readable program code means fur- 
ther comprising: 

means for displaying said moving picture infor- 20 
maii on and sound information relating to said 
moving picture information on said display 
device in a common time axis; and 
means for displaying a result of editing of the 
sound information commanded according to a 25 
user's command for adjusting parameters relat- 
ing to sounds of the sound information in said 
editing position. 
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